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ABSTRACT 

This paper describes the experiences of researchers 
conducting research using an online course for the professional development 
of middle school mathematics and science teachers. The paper also offers 
recommendations to developers of course management systems (CMS) . The online 
course focused on strategies that promoted gender equitable pedagogy and 
practice to classroom instruction. The paper suggests that if a researcher, 
or someone knowledgeable about what researchers require and do with data, 
referred to in this paper as "researcher's voice," is made an integral part 
of the creative team of a CMS, researchers would be more likely to trust the 
accuracy of the data yielded by a CMS and rely on the stability of a system 
to collect and present data in its original format. The paper highlights 
three experiences that speak to this need. The first one deals with loss of 
data, the loss of presurvey data collected through an online survey but not 
stored, for unknown reasons, in the system* s database. The second concerns 
the accuracy of course statistics offered by the system, and the third 
discusses the importance of keeping the original formatting and text of 
threaded messages where the data is exported to an external site. (SLD) 
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Abstract 



The use of Web-based course management systems (CMS) for the design and 
delivery of online courses is increasing. While these kinds of online courses provide 
course statistics, teacher grade books, and other tools and features that yield data that can 
be used for research and evaluation purposes, these data are not necessarily accurate or 
reliable. At the same time, the system for collecting and storing these data may not be 
stable, leading to loss of data. Further, discussion board threads, when exported to 
another site, do not always follow the order of the threaded messages and include junk- 
text (including meta-tags); thus, requiring a great deal of cleaning up and reformatting of 
the text prior to conducting any kind of analysis. 

There are various kinds of voices that are integral to the development and 
implementation of a course management system. These voices include those of a 
programmer, art director, and others. What seems to be missing is the researcher’s voice, 
that is, someone knowledgeable about what researchers require and do with data. I posit 
that when the creative team of the CMS also includes a researcher’s voice, there is better 
likelihood that the data generated by this system are accurate, reliable, and reflect the 
original format when the data was first presented, as well as the system for collecting and 
storing data is made more stable. 

This paper highlights our experiences when we conducted research using an 
online course for the professional development of middle school math and science 
teachers. It also offers recommendations to CMS developers. The online course focused 
on strategies that promoted gender equitable pedagogy and practice to classroom 
instruction. 
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Introduction 



The use of the World Wide Web as an educational platform is increasing. We see 
a plethora of online courses from various academic institutions, many of which are being 
offered towards either an undergraduate or graduate degree. We also see online courses 
geared for professional development of teachers and employees, some offering 
professional development points or college credit. 

These online courses are usually delivered through course management systems 
(CMS) such as WebCT and Blackboard. Most CMS include tools and features for 
instructors and participants to use. For instance, there are templates for instructors to 
upload their course syllabus and readings, as well as post announcements. There are 
other templates that instructors and participants can avail of to design their individual 
homepages. Moreover, there is a discussion board where instructors and participants can 
post their messages regarding the issues raised in the readings and other relevant topics, 
as well as for participants to post their projects. There is also email capability for public 
as well as private correspondence. Further, with such course management systems, 
instructors are able to design and implement surveys, as well as collect the data from 
these surveys. In addition, instructors are provided with a grade book they can use to 
record and evaluate student progress. 

While these course management systems include research and evaluation* 
instruments such as surveys, these programs are not necessarily yielding accurate data 
and the system for collecting the data is not always stable. At the same time, when data, 
such as the messages in the discussion board, are exported onto an external database, the 

' The term “research” will be used in this paper to include research and evaluation; and 
“researcher” to include researcher and evaluator. 
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formatting of the threaded messages is usually lost and the imported data contain junk- 
text (such as meta tags) which necessitates cleaning up the data before any coding and 
eventual analysis could be done. This was our experience while doing research on an 
online course on gender equity geared for the professional development of middle school 
math and science teachers. This paper discusses these experiences and the lessons we 
learned, underscoring the need for a researcher’s voice within the creative team of the 
course management system, and offers suggestions to CMS developers, who, as a team, 
have collective and individual input to the final design of these systems. 

Researcher’s Voice 

Computer-based software applications are usually created by a team composed of 
designers and programmers (Bowers, 1998; Matthis, 1997, 1999; Provenzo, 1996). This 
creative team has individual and collective voices such as those of user interface 
designers, instructional designers, and programmers. Matthis (1999) referred to voice as 
the “expression of thoughts, feelings, desires, opinions, will, personality, philosophy, 
values and choices, spoken or unspoken, of the software designer(s) or programmer(s), in 
the software program” (p. 3). Similarly, Web-based course management systems are also 
developed by a creative team composed of several voices. 

I posit that, if a researcher, or someone knowledgeable about what researchers 
require and do with data — which I refer to as researcher’s voice — is made an integral part 
of the creative team of a CMS, researchers would be more likely to trust the accuracy of 
the data yielded by a CMS and rely on the stability of the system to collect and present 
the data in its original format. Researchers would be more willing to conduct a study that 
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is based on quality data from online courses. Thus, if the researcher’s voice was also 
present and serious attention paid to it, the CMS could be made to ensure that only 
accurate and reliable data are available to both the instructor and anyone interested in 
doing research with the data. Researchers rely on these kinds of data and need to feel 
confident about the results of their analyses from such data. 

The need for this researcher’s voice was made apparent in the course of our 
research. As a brief background, our research investigated how participants (mostly 
middle school teachers of math and science from around the U.S. and other parts of the 
world) of an online course on gender equity learned about gender equity and whether 
they have integrated equitable pedagogy and practice into their classroom instruction as a 
result of taking the course. Our sample included participants who agreed to be part of our 
research and who participated in one of the four offerings of the online course given by 
our partner-organizations: the Women’s Educational Equity Act Resource Center at 
Education Development Center, Inc., TERC, and the Eisenhower National 
Clearinghouse. Our research, called “Gender Equity in Math and Science” (GEMS), was 
funded by the National Science Foundation. 

Our Experience 

Before I start a discussion of our experiences with the Web-based CMS, I want to 
underscore that this paper is not meant to lambaste the CMS used for this online course . 
Rather, I would like to use our experience to demonstrate the need for CMS developers to 

^ Based on our conversations with other users of various kinds of course management systems, our 
narratives and the lessons we have learned are fairly similar to their own experiences. Thus, we are not 
mentioning the CMS used for the online course as we would rather focus on the issues being raised in this 
paper instead of on a particular CMS. 
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pay more attention and interest in ensuring that a researcher’s voice becomes a major part 
of their creative enterprise. 

I highlight three experiences that speak to this need. The first one deals with loss 
of data and thus opens up the question of stability of the system. The second focuses on 
the accuracy of course statistics offered by the system. The third discusses the 
importance of keeping the original formatting and text of threaded messages when the 
data is exported to an external site. 

When we began our research project, we used the survey template of the CMS to 
design a pre- and post-survey. We asked the participants of our first offering of the 
online course to complete the pre-survey prior to actively engaging with the online 
course. Almost all of the participants submitted their responses, and our expectation was 
that the data were going to be deposited to a database that we could then access within 
the course management system. By the time we accessed the database, we could not find 
the survey responses. I telephoned the technical assistance/help desk of the CMS 
developer and was told that they would look into our problem. The technical assistant 
(TA) informed me at that time that “that should not have happened.” I was optimistic 
that they would come back to me with the data intact. However, this was not the case. 
While the TA apologized profusely for losing our data, there was nothing else he and 
other staff members could do to retrieve the data. 

Thus, we lost all of our pre-survey responses. We could not ask our participants 
to complete our pre-survey again since, by the time we found out about the lost data, they 
had already started taking the online course. Because of this experience, we decided to 
hire an outside consultant to design a Web-based survey template and database for us and 
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who made sure that we could access the database with no time restrictions. I told the 



consultant that we needed a very stable system for our research and shared our experience 
with him. Hiring such a consultant was not anticipated and therefore affected our project 
budget. From then on, all of our participants, including those who were part of our three 
other offerings, used our consultant’s website to view and answer the other surveys that 
were part of our research. In addition, loss of data affected the design of our research. At 
the start of our research, we were keen to analyze pre- and post-survey conditions of the 
participants and make the results of that analysis an integral part of our study. Now, we 
had to rethink our research design, and one of the changes to our methodology was the 
use of case studies. Thus, we had to narrow our official sample from many participants 
to a selected few. 

While ensuring the stability of the system is crucial for any CMS, the availability 
of accurate data is equally important. The CMS for the online course offered course 
statistics that reflected the number of times participants entered an area within the course 
(e.g., readings, discussion boards), at what time of day and day of the week they accessed 
these areas, and the totals for each participant. While this feature is useful, it does not 
necessarily convey precisely what each participant did. Some specific examples are of 
value here. 

There were three main areas within the course itself, namely, communication, 
main content, and student. Each area had several features and tools. The communication 
area, for instance, included the discussion boards and email system that instructors and 
participants could use. The main content area had course readings, case studies, and web 
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links to additional resources. The ability to design individual web pages and similar 
student tools were available in the student area. 

When we collected all the statistics for the first two sessions of the first offering 
of the course, we noticed that there were numbers that did not seem to make sense. On 
one occasion, a participant had a total of 156 hits in a single day, another individual had 
140, and two others had 0 and 11. We then started to ask ourselves where those numbers 
came from. Could it be from the number of clicks a participant had when s/he was in a 
certain area? If so, what does that tell us about what the participant did? Unless the 
individual with 156 hits for a particular day spent a huge amount of time navigating the 
course, that number seemed to make no sense. Given that participants were expected to 
spend three hours on average for each nine-day session, this number seemed too high for 
a single day. 

Because the numbers we saw were suspect, I called the course management 
system’s TA to find out what these numbers were supposed to capture. I was told that 
they reflected the number of times that an individual entered a particular area, regardless 
of what they did in that area. So it is possible that a participant clicked on a case study 
within the main content area in Session 1 ten times; however, we do not know if that 
person really did the reading. The only conclusion we can draw from this statistic is that 
a participant clicked several times within that area; nothing more substantive can be 
gleaned from these data. So, are these data helpful at all to researchers? Perhaps for 
some, but definitely not for our purposes. 

To find out for myself how accurate the data offered by the course statistics 
feature of the CMS were, I tracked myself while I entered the different course areas. I 
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carefully wrote down where I went and what I did. I counted that I entered the 
communication area twice: to read the description of the session’s discussion and one 
posted message. In the main content area I read descriptions for Sessions 1 and 2, as well 
as the readings and case studies for both sessions. I counted 1 1 clicks. In the student 
area, I clicked once on a user tool that allowed me to write some personal information. 
Thus, I counted a total of 14 clicks. However, when I accessed the course statistics 
feature and highlighted the results for my activity, I found a discrepancy in my numbers 
and what the system gave me. I was given a total of 21 hits, 5 for the communication 
area, 14 for main content area, and 2 for student. I wondered if I should have counted the 
number of times I clicked on the other buttons within that area additionally, including the 
‘Back’ button and close box. To gain some insight into the discrepancy, I telephoned the 
TA. He tried to give me a logical explanation, but there was really nothing he could tell 
me that could help me to understand what these numbers revealed. 

Thus, I could not rely on the available data to make any conclusive statements 
about what the participants did within the course itself. As a researcher, I could not 
afford to have inaccurate or unexplainable data. The results of my analysis would be 
suspect as well. 

On a different note, we exported the messages from the discussion board onto an 
external database so we could conduct an analysis of the discourse. We expected the 
order of the threaded messages to be reflected in the order of the exported text messages. 
In other words, all the responses to a posting would be grouped together. They were not. 
Instead, the exported data followed the order of the postings as though there were no 
threads. So, in order to follow the threaded interaction, we had to rely on a hard copy 
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that showed the order of the threads and the messages for each thread. Further, the 
exported data contained meta tags and other junk- text that were not part of the original 
postings. In light of this, we had to manually delete these junk-text, as any global 
replacement of text was unfeasible. 

The challenges we faced to collect and understand the data yielded by the course 
management system of the professional development offering were many. However, 
they provided us with good insights into how course management systems can be 
developed that support research undertakings. 

Conclusion and Recommendations 

It would be reasonable for researchers like us to want a Web-based CMS for our 
online courses that could provide robust data that are both useful and accurate. 

Moreover, we would like to expect that we could at least access the database at any point 
in the course of our research — we rarely find humor in lost data. At the same time, we 
would like the original formatting of data to be preserved while transferring the data onto 
an external database so we can conduct further analysis of the data, including the use of 
software programs designed specifically for either quantitative or qualitative analysis. 

We offer a few recommendations to CMS developers as they engage in the 
creation of their programs. First, when putting together the creative team, make sure that 
someone knowledgeable about conducting research is made an integral part of the team. 
This researcher’s voice is important for any education-related activity. Second, pay 
attention to the bugs and other technical problems that could result in the loss or 
reformatting of data. If at all possible, come up with a back-up database system that can 
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restore responses to surveys. Even though only certain data might be recovered, in many 
cases this is better than losing all the data. Moreover, find a way to ensure that meta tags 
and other programming codes are simultaneously eliminated while in the process of 
exporting data. Third, while course statistics are an important feature, it would benefit 
educators and researchers alike if these statistics tell more than just the number of clicks a 
participant had while navigating through the course. Course statistics could also include 
accurate data on what participants did for each session of the course, and not just an 
accumulative account of what they did within the course. With this session-by-session 
information, instructors would be able to monitor student participation and progress, and 
could provide certain interventions to help a student stay on track such as personal email 
to check-in with a student, provide some hand-holding when necessary, and motivate a 
student to stay with the program. These interventions, done early on, could help reduce 
the drop-out rate. Fourth, when technical assistants or help desk consultants answer the 
phones, it would be helpful to those on the other side of the line if these individuals were 
able to give some clear explanation as to why certain things are happening with the CMS 
and the data. It is frustrating for educators and researchers to be in the dark as to why 
their data are lost, for example, or what certain numbers might tell them. 

We understand that CMS continue to be updated, with some of the kinks in the 
system reduced, if not eliminated. I believe that we have gone a long way in designing 
Web-based user interfaces for online courses. I foresee that if CMS developers pay 
closer attention to our experiences and the hard lessons we have learned, the education 
and research fields will be better served. This mutual support between CMS developers 
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and education/research practitioners could lead to a major advance in the way we use 
Web-based tools for teaching, learning, and research. 
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